Add ResampleConcat; harden Resample against non-monotonic reference clocks#161
Merged
Conversation
Resampling one stream onto another stream's clock and concatenating the result is a common pattern (e.g. merging two acquisition hubs with slightly different effective rates). The existing Resample -> Merge graph has two problems when the reference clock misbehaves: 1. A backward jump in the reference clock leaves every incoming reference value behind the resampler's high-water mark, so it stops producing output and the whole graph seizes. 2. Even when the resampler holds back or drops reference samples, the raw reference stream feeding Merge.INPUT_SIGNAL_A is no longer sample-aligned with the resampled stream (Align does a one-time alignment, then reads in lockstep), silently concatenating mismatched samples. Changes to resample.py: - output_reference setting: buffer reference data and emit it gathered onto the exact grid the signal was resampled onto, via a new OUTPUT_REFERENCE stream. A downstream Concat can combine the two with no second alignment. - reference_reset_after_chunks: detect a sustained backward clock jump and re-anchor the high-water mark (with a RuntimeWarning) so output resumes instead of seizing. Self-correcting jitter is dropped, keeping output monotonic. Set to inf to restore the previous behaviour. - push_reference now accepts a CoordinateAxis reference (no .gain attr). New resampleconcat.py: ResampleConcat fuses ResampleProcessor (output_reference=True) and ConcatProcessor into a single linear two-input unit, aligned by construction -- no Align, no diamond graph. Reuses all interpolation and axis/attr-merge logic from resample.py and concat.py. Tests: unit coverage for the new resample features and the composite, plus integration tests covering the healthy graph, the seize when recovery is disabled, recovery via OUTPUT_REFERENCE, and the composite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Combining two acquisition streams that share a nominal rate but drift slightly (e.g. two hubs at ~30 kHz with 0.1–0.5% effective-rate mismatch) is done by resampling one stream onto the other's clock and concatenating along the channel axis. The existing
Resample → Mergegraph has two failure modes when the reference clock misbehaves:MERGE.INPUT_SIGNAL_Ais no longer sample-aligned with the resampled stream —Aligndoes a one-time offset alignment then reads in lockstep, so it concatenates mismatched samples.Changes
resample.pyoutput_referencesetting + newOUTPUT_REFERENCEstream: buffers the reference data and emits it gathered onto the exact grid the signal was resampled onto, so a downstreamConcatcan combine the two with no second time-alignment (aligned by construction).reference_reset_after_chunks(default 3): detects a sustained backward clock jump and re-anchors the high-water mark with aRuntimeWarninginstead of seizing. Self-correcting jitter is dropped, keeping output monotonic. Set toinfto restore the previous behaviour.push_referenceCoordinateAxis fix: no longer unconditionally readsax.gain(which crashed on a per-sample-timestamp reference).resampleconcat.py(new)ResampleConcatfusesResampleProcessor(output_reference=True) andConcatProcessorinto a single linear two-input unit — noAlign, no diamond graph. Reuses all interpolation logic fromresample.pyand all axis/attr-merge logic fromconcat.py; only the orchestration is new.Tests
tests/unit/test_resample.py: CoordinateAxis reference,output_referencegrid-alignment, reset recovery, and reset-disabled-stalls.tests/unit/test_resampleconcat.py(new): two-rate combine with numeric provenance check, no-signal, reset recovery.tests/integration/ezmsg/test_resample_merge_system.py(new): healthy graph; seize when recovery disabled (documents the original bug); recovery viaOUTPUT_REFERENCE; composite recovery.Full suite: 3596 passed, 5 skipped locally.
Notes
Output across a genuine reference clock reset is necessarily discontinuous — the hardening is a guardrail (don't hang, warn loudly) rather than a substitute for sanitising reference timestamps upstream.